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ABSTRACT 

In  the  present  paper  we  consider  the  following  stochastic 
control  problem:  minimize  the  average  expected  total  cost 

T 

J(x,u)  =  lim  inf  i  E^  [  [4»  (5  )  +  |u.(C)|]dt 
T-*-*  1  x  J  o  z  z 

subject  to  d£t  =  ut(£)dt  +  dwt ,  S  Q  =  x;  |u|  <  1,  wt  Wiener, 
with  all  bounded  by  unity  and  measurable  functionals  on  the  past 
of  the  state  process  {5g;  s  <  t}  admissible  as  controls.  It  is 
proved  that  under  very  mild  conditions  on  the  running  cost  function 
$(•)  the  optimal  law  is  of  the  form 

u*U)  -  -1,  Ct  >  b  * 

•  0,  Kt|  <  b 

=  1,  Kt  <  -b. 

The  cutoff  point  b  and  the  performance  rate  of  the  optimal  law  u* 
’are  simultaneously  determined  in  terms  of  the  function  <J>  ( • ) 
through  a  simple  system  of  integrotranscendental  equations. 


OPTIMAL  STATIONARY  LINEAR  CONTROL  OF  THE  WIENER  PROCESS 


1.  INTRODUCTION 


*51  In  this  paper  we  consider  the  problem  of  stationary  control  of 
the  stochastic  differential  equation,  d£  =  u.  (^)dt  +  dw  ;  £  =  x, 

_  L  L  L  U 

where^  (m -n-4^;  t  >  0)  is  a  Wiener  process  on  an  underlying 
probability  space  .  (-ft 

’  *t) 

Two  kinds  of  cost  are  involved  in  this  problem.  First,  one 
Pk.  UeX  ~  )  7.  f  *f„  Sub  ^ 

pays  ^  ^  per  unit  time  for  being  in  the  wrong  state  , 

where  <J>  ( • )  is  a  suitable  cost  function  to  be  described  later; 

secondly,  one  pays  I  Per  unit  time  for  using  the  control 

1  u  i'.-v  ? 

law  The  control  problem  is  to  choose  a  law  ut(£)  as  a  non- 

anticipative  functional  of  the  solution  process^  (£t)  with  values 
in  the  bounded  interval  [-1,1],  so  as  to  minimize  the  average 
expected  total  cost. 

It  is  proved  that  the  optimal  law  can  be  explicitly  described 

and  its  performance  characterized  in  terms  of  the  cost  function  » 

<)>(•)•  The  method  consists  in  first  restricting  attention  to  an 

1 

important  subclass  of  admissible  control  laws,  namely  those  giving 
rise  to  an  ergodic  solution  process  (£t).  A  process  is  said  to 
be  ergodic  if  it  admits  a  unique  invariant  distribution.  The 
optimal  law  u*  in  this  subclass  can  be  obtained  by  using  a 
dynamic  programming  approach,  similar  to  that  of  Wonham  [11]; 
it  turns  out  that  u*  is  of  the  form 


(1.1) 


uj(£)  =  -sgn  £t,  |£t! 

*  o.  IU  5  b 
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where  b  is  a  positive  constant  that  can  be  characterized  in  terms 
of  the  function  $  ( ■ ) .  Secondly,  the  law  u*  is  proved  optimal 
against  any  possible  nonanticipative  law  u  whatsoever. 

The  result  (1.1)  is  the  natural  and  expected  one;  it  says  that 
the  best  policy  is  to  push  with  full  force  in  the  negative 

direction  if  it  is  too  positive  and  in  the  negative  direction  if  it 
is  too  negative,  while  refraining  from  any  action  if  is  in¬ 

side  a  "dead-zone"  [  - b , b  ] .  The  appearance  of  the  latter  is  a  con¬ 
sequence  of  the  running  cost  |u|  on  the  control,  of  the  fact  that 
the  control  is  "expensive".  Were  such  a  cost  absent,  it  is  fairly 
obvious  -  ane  easily  probable  by  using  the  methods  of  the  present 
paper  -  that  the  optimal  policy  would  be  described  by  the  "bang-bang' 
law:  -sgn 

Among  previous  works  on  the  topic  of  stationary  control  of 
systems  driven  by  a  Wiener  process  we  cite  those  of  Wonham  [11]  and 
Kushner  [8].  The  scope  of  both  was  severely  restricted,  however, 
in  that  they  allowed  only  those  laws  that  generate  an  ergodic  solutio 
process  (actually,  only  a  subclass  of  these  was  considered). 

2.  FORMULATION 

Consider  the  space  fi  =  C^g  jj  of  real-valued,  continuous 
functions  on  [0,T],  for  some  T  >  0.  Let  (5^)  denote  the  family 
of  evaluation  functionals  on  Cjq  and  5*^,0  <  t  <  T  the 

o-field  of  subsets  of  Tj  generated  by  s  <  t). 

Consider  also  the  o-field  Jt  of  subsets  M  of  [0,T]  *  C^q 
having  the  property  that,  for  any  t  £  [0,T],  belongs  to 
and  that  each  ^-section  of  M,  £  £  Cjq  jj ,  is  Lebcsgue  measurabl 

A  function  g  defined  on  [0,T]  *  C[0,TJ  is  ^-measurable  if 
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and  only  if  g(t,*)  is  ^.-measurable,  for  each  t,  and  g(*,£)  is 
Lebesgue  measurable,  for  each  5. 

Definition  2.1;  Let  the  control  measure  space  be  the  interval 
[-1,1]  with  its  Borel  sets.  An  admissible  nonanticipative  control 
law  u  is  a  measurable  function  u:  ([0,T]  *  jyjt)  [-1,1]. 
The  class  of  all  such  control  laws  is  denoted  by  °k . 

For  any  control  law  u  £  and  any  x  £  1R,  a  weak  solution 
(St)  to  the  stochastic  differential  equation 


(2.1) 


d?t  =  ut(5)dt  +  dwt;  0  <  t  < 


(2.2) 


S  '  x 


is  constructed  as  follows:  one  starts  with  the  probability  space 
(fi,  _S^,P),  where  P  is  Wiener  measure  on  Si  -  Tj  .  Corre¬ 
sponding  to  each  law  u  £  ^  and  each  initial  position  x  £  1R,  the 
new  measure 


(2.3)  P“(d«)  =  exp 


ut(Ociwt 


o  u?«>d*J- 


\ 


P  (d  <*>) 


is  constructed  on  (n ,  5j) ,  where  (5^)  is  the  process  defined 
by  £  *  x  ♦  w  ;  0  <  t  <  T.  According  to  Girsanov  [5],  P^1  is  a 
probability  measure  on  (q  ,  _^j)  and  the  process 


(2.4) 


-  A 
Wt  =  Wt 


u  (Ods 


'  ' x  ■  IS 


(5)ds 
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.uuuu  i  iiiiuu  m  hi  m  l  mm  . . - .  1  * 

is  a  Wiener  process  on  (fl,  Equation  (2.4)  is  an  equival' 

way  of  saying  that  the  process  (5^),  =  x  +  w^. ;  0  <  t  <  T 

satisfies  the  stochastic  differential  equation 

(2.1) '  <15 1  =  ut(5)dt  +  dw  ;  0  <  t  <  T 

(2.2)  5q  =  x 

on  (fi,  J*^,P^).  All  processes  involved  here  are  adapted  to  the 
underlying  family  (  &  )  of  sub-o-fields  of  j£_.  The  process 
(5t)  is  called  a  weak  solution  of  (2.1) ’  -  (2.2)  because  by  con¬ 
struction  °(^s'>  s  <  t)  c  s  <  t),  though  not  necessarily  the 

other  way  around.  Such  a  solution  is  known  to  be  unique  in  the 
sense  of  the  probability  law;  see  Liptser  and  Shiryayev  [9]. 

Now  consider  a  function  4>  :  ]R  -*■  ]R+  which  is  even,  convex 
( 2 1 

piecewise  Cv  ,  monotonicnlly  increasing  to  infinity  on  x  >  0, 
and  satisfying  an  exponential  growth  condition: 

(2.5)  4>  )  =  0(ea^)  as  |x|  -*•  °° ,  some  0  <  a  <  2. 

\ 

The  optimal  control  problem  can  now  be  formulated  as  follows: 
choose  a  law  u  £  ^  for  which  the  limit 

J(x,u*)  =  lim  I  P.“*[  («(5t)  +  |  u*  (5)  |  )dt 

T-  JoT 

exists  for  all  x  £  IR,  and  which  minimizes  the  average  expected 
total  cost  rate 
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T 

(2.6)  J  (x,u)  =  lim  inf  \  [  [4»(Ct)  +  (uJ^lJdt 

J-KO  1  X  J  0  Z  Z 

of  starting  at  place  x  and  using  control  law  u,  for  all 
(x,u)  €  pRx  *2^]  .  denotes  expectation  with  respect  to  the 

probability  measure  P^1  introduced  in  (2.3). 

3.  SUMMARY 

In  Section  4  we  briefly  study  the  important  subclass  of  feed¬ 
back  (Markov)  admissible  control  laws.  It  is  pointed  out  (and  in 
the  special  case  of  time-homogeneous  feedback  laws,  proved)  that 
for  such  controls  the  stochastic  differential  equation  (2.1)-(2.2)  r 

the  system  can  be  solved  in  the  strong  sense. 

In  Section  S  we  consider  a  subclass  of  time-homogeneous  feed¬ 
back  laws  that  give  rise  to  an  ergodic  solution  process.  Asymptotic 
properties  of  those  processes,  such  as  existence  of  a  unique  invaria 
measure,  laws  of  large  numbers  and  ergodicity  of  their  distributions 
are  discussed. 

•fe  * 

The  optimal  law  u  in  the  abovementioned  subclass  is 
discerned  in  Section  6  and  it  is  proved  that  u*  is  of  tile  form 
(1.1).  Both  the  cutoff  point  b  and  the  asymptotic  performances  ^ 
rate  of  u*  are  characterized  in  terms  of  the  cost  function  4s  ( • )  , 
through  the  system  of  integrotranscendental  equations  (6.3),  (6.4). 
The  method  proceeds  by  constructing  a  solution  to  the  "asymptotic" 
version  of  the  Bellman  equation  of  dynamic  programming  (6.2). 

Finally,  the  asymptotic  performance  of  the  law  u*  is 
compared  against  that  of  any  admissible  nonanticipative  control  u 
in  The  result,  proved  in  Section  7,  is  that  u*  is  actually 
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optimal  in  the  (largest  possible)  class  <2r.  The  idea  employed 
here  is  to  first  compare  the  performance  of  the  control  laws  ovei 
finite  time  intervals  [0,T]  and  then  pass  to  the  limit  as  T  -*• 

\ 

4.  MARKOV  LAWS  AND  STRONG  SOLUTIONS 

Definition  4.1.  Suppose  there  exists  a  measurable  function 
Y:  1R  x  [0,T]  [-1,1]  such  that  the  nonanticipative  law  u  6  ^ 

can  be  represented  in  the  form 

(4.1)  ut(5)  =  Y(5t,t),  an>"  5  £  C[o,T]’  0  5  1  5  T- 

Then  u  is  called  an  admissible  Markov  law.  The  class  of  all 
such  laws  will  henceforth  be  denoted  by  ssf\  obviously  <=  cfy . 

For  laws  in  _i/  the  stochastic  differential  equation 

(4.2)  d£t  =  Y  (5 1 » t )  dt  +  dwt,  =  x 

is  known  to  possess  a  pathwise  unique,  strong  nonanticipative 
solution,  in  the  sense  that  the  solution  is  adapted  to  the  Wiener 
process:  a(5s;  s  <  t)  c  °(ws',  s  <  t),  0  <  t  <  T;  see  Zvonkin  [12] 

Definition  4.2.  Consider  the  subclass  of  a/  consisting  of 
those  admissible  nonanticipative  laws  u  for  which  there  exists  a 
measurable  function  a :  1R  -*■  [-1,1],  such  that 

(4.3)  u  (?)  =  a  ( £  .  )  ,  any  C  €  C  ff)  ,  0  <  t  <  T. 
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Such  laws  u  are  called  admissible  time -homogeneous  Markov  laws 
and  their  class  is  denoted  by 


For  laws  in  &  one  can  easily  construct  the  (pathwise  unique) 
strong  solution  to  the  stochastic  differential  equation 


(4.4) 


d£t  =  a(5t)dt  +  dwt,  0  <  t  <  T 


Indeed,  consider  the  function 


(4.5) 


B(x) 


■x 

exp{  -  2 

0 


fy 

a (z) dz } dy ; 

0 


x  £1R 


which  is  continuous,  strictly  increasing  and  satisfies  the  equation 
3"  +  2a3'  =  0.  The  function 


(4.6)  o(x)  =  6'  (  3~ 1  (x) )  ;  x  £  1R 

I 

is  Lipschitz  continuous,  as  can  be  checked  by  simple  calculus. 
Therefore  the  stochastic  differential  equation 


(4.7)  dCt  =  o(4t)dwt;  0  <  t  <  T 

•  (4.8)  C  =  6(x) 

has  for  any  x  €  IR  a  pathwise  unique  solution  (Ct)  on  the 
probability  space  (n,J?j,P),  strong  in  the  sense  that 
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a ( £ s ;  s  <  t)  c  o(ks ;  s  <  t),  any  0  <  t  <  T,  according  to  Itb's 
classical  theory;  see  for  instance  Gihman  and  Skorohod  [4].  Denote 
by  {Q,  >Pg(x j 1  the  corresponding  time-homogeneous  Markov 

process 

The  process 


(4.9) 


1 

«t  -  e-'tcp 


is  now  well  defined,  and  an  application  of  Ito's  rule  gives 


i  i  B" ( e~ 1  cc,. ) ) 

rj -  dt  -  \  - — 

)  A  c  r  \  \  L  ^ 


1  8’  1  2  (  B*  (B' Vt))r'> 


o  (Ct)dt 


=  a(£.)dt  +  dw. 
t  t 


So  (f.^.)  satisfies  both  the  equation  and  the  initial  condition 
in  (4.4)  and  because  it  is  a  bijection  of  (t^)  pointwise  in 
time: 


o{,?;  s  <  t}  =  oUs  ;  s  <  t)  c  o{w? ;  s  <  t }  ,  0  <  t  <  T 


i.e.  (£^)  is  a  strong  solution  to  (4.4).  The  corresponding  time- 
homogeneous  Markov  process  is  denoted  by  { Q ,  5y,  yt»St,r”). 

5.  SOME  ERGODIC  THEOREMS 


Introduce  the  function  G(x)  = 


x 


— i— ! —  »  a ( ’ )  as  in  (4.6), 
-°°  o^  ( z ) 


and  consider  the  subclass  feT  of  tf,  consisting  of  those  laws 


a 


i 
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u,  ut(£)  =  a(£t)  for  which 


(5.1) 


G(») 


da 

°2(z) 


00  ty 

exp{2  a(z)dz)dy  < 
-co  Jq 


recall  also  the  processes  (£t),(St)  of  the  preceding  section, 
corresponding  to  this  law.  According  to  Gihman  and  Skorohod  [4; 
§18],  the  probability  distribution  is  ergodic  for  the 

Markov  process  (fi,  .5^,  ^;>^t,Pz^  in  the  sense  that  the  following 
are  true: 


Fact  1.  Positive  Recurrence:  The  stopping  times  f =  - 

inf(t:  Qt  =  y)  are  well  defined  and  a.s.  finite  for  any  z,  y  €  1R; 

besides , 

(5.2)  Ez(Tzy)  <  G(»)(2+|z-y|) |z-y| . 


Fact  2.  Invariance  of  the  Probability  Distribution  Function  G(*)/G 
For  any  0  <  t  <  T, 


(5.3) 


00 

P“{5  <  y)dG(z)  =  G(y) ,  y  €  ]R. 

-  00 


Fact 

that 


,3.  Law  of  Large  Numbers:  For  any  Borel  function 

oo 

|f(y)|dG(y)  <  ,  we  have 

,  -  oo 


f(-) 


such 


•  (5.4) 


lim  i 

T-*oo  1 


T 


f(Ct)dt  = 
0  1 


1 

G  (“) 


f  (y ) dG (y )  ; 


a .  s 


(Pu) 

z 


and 


1  u 
L  (E“), 


any 


e  JR. 
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"non** 


Fact  4.  Ergodicity  of  the  Distributions:  For  any  function  f(-) 
as  above, 


(5.5) 


.CO 

lim  E^f(C  )  =  rTTooT*  f (y ) dG (y )  ,  any 

^  ->00  (  '  J  „ OO 


z  =  5, 


As  a  consequence: 


lim  PllU  <  y} 
t-*°°  z  t 


(■(y) 

gFT 


y  €  ]R. 


It  follows  from  the  properties  of  the  function  f3 ( • )  intro¬ 
duced  in  (4.5)  that  the  limiting  distributions  of  the  processes 

(Ct)  and  (£>t)  exist  simultaneously.  Consequently,  the  probabili 

F  f  •  ) 

distribution  function  y  ,  where 


(5.6) 


r(x,  S  g(»(x),  - 


•X 

exp{  2 


y 

a(z)dz)dy, 

0 


is  invariant  for  the  Markov  process  5^  ,S t , P^} .  The  ergodi 

properties  of  the  latter  can  be  read  off  from  those  of  the  (4^) 
process  : 

l 


(5.3)' 


CO 

PX  5  y)dF(x)  =  F  ( y )  ,  any  0  <  t  <  T,  y  e  1R. 


(5.4) ’ 

(5.5)  ' 


1 

T 


T 


f(^f)dt 
0  C 


— y 

T+<» 


i 

F  (”) 


CO 

f(y)dP(y),  a . s . 

-  CO 


(P“)  and  L1 (I 


lim  El‘f(5t) 
t->°°  1 


1 

FT^T 


OO 


f(y)dF(y) , 


any  Borel  function  f(')  such  that: 


11 


I  f  (y) j  di:(y)  <  «  , 

.  -  oo 


any  x  =  £  Q  €  IK. 


Proposition  5.1:  For  any  law  u  £  ut  (£)  =  a(5t),  the  corre 
sponding  solution  process  =  %  of  the  system  equation  (4.4) 
is  a  strongly  Feller  process,  pocessing  a  unique  invariant 
probability  distribution  FU  ( * ) /FU  (°° )  ,  Fu(x)  =  F(x)  as  in  (5.6), 
for  which  (5 . 3) '  -  (5 . S) ' hold . 

Proof .  All  that  remains  to  be  proven  is  the  strong  Feller  property 
and  the  uniqueness  of  the  invariant  distribution,  and  it  suffices 
to  do  both  on  the  (4^)  process.  The  latter  is  indeed  strongly 
Feller,  since  (5.1)  implies  a  fortiori:  o2(z)  >  a2,  all  z  £  1R, 
some  o2  >  0;  see  Wonham  [10).  On  the  other  hand  (£  t)  is  recurrc 
and  positive,  by  (5.2).,  For  such  processes,  Khas'minskii  [7] 
proves  the  existence  of  a  unique  invariant  distribution,  Q.E.D. 


Definition  5.2.  For  the  constant  a  of  (2.5),  0  <  a<2,  let 

be  the  subclass  of  r V  consisting  of  those  laws  u, 
u  U)  -  a  (t»  t )  for  which  \ 

CO  °° 

(5.7)  calxklFu(x)  =  e°t^x^exp{2|'a(z)dz}dx<00. 

.  -  oo  -oo  /  Q 

It  is  evident  from  (5.4)’  and  the  assumption  (2.5)  that,  for 

any  u  £  &  : 

a 
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(5.8) 


J(u) 


J(x ,u) 


1  im 

'I'-kCC 


i:u  f 
x  ir 


[♦(St) 


|ut(5)|]dt 


1 

nU  ,  V 

F  (=o) 


[♦  (y) 


+  |a(y)|]dFu(y), 


any  x  =  4  Q  6  IR. 

8.  THE  OPTIMAL  LAW  IN 

Introduce  the  function 


(6.1) 


,  .  A 

c(p)  = 


min  (up+ | u | ) 

I  u|  <1 


=  1  *  I P I  ,  I P I  >  1 

=  0,  | p |  <  1. 


Our  objective  is  to  find  a  positive  constant  X  and  a  function 
v(x),  twice  continuously  differentiable  on  IR  and  0(ea^x^)  as 
|x|  -*■  09 ,  with  0  <  a  <  Z  as  in  (2.5),  satisfying  the  Dynamic 
Programming  equation 

(6.2)  X  =  ~  vxx(x)  +  c(vx(x))  +  <p  (x)  ,  x  e  IR. 

We  start  with  a  preliminary  result. 

Lemma  6.1.  Under  the  assumptions  on  the  running  cost  function 
(f>C*)  made  in  section  2,  there  exists  a  unique  solution  ( A  , b )  to 
•  the  pair  of  equations 


fb 

Xb  -  <J>(s)ds  = 
1  0 


(6.3) 


1 

7 


(6.4) 


X  =  2  I  e  “*'♦  (b  +  s )  ds 
i  r> 


Proof.  It  suffices  to  prove  that  equation  H(x)  =  0, 


(6.5) 


H(x) 


r00  r  x 

=  2x  e  2s<f>(x  +  s)ds  -  4>(s)ds  -  i  , 

Jo  J  n  L 


has  a  unique  solution  b  on  1R  .  Indeed,  H(0)  =  -1/2  and 


H' (x)  =  2  f°e  2s[<j>(x+s)  -  $(x)]ds  +  2x  f  e  2sp'(x+s)ds  >  x<J>'(x),  x  >  0. 
Jn  h 


Clearly,  H(x)  -*■  °°  as  x  so  there  exists  a  unique  number 

b>  0 ,  such  that  H(b)  =  0,  Q.F..D. 


The  constants  (A,b)  being  as  in  the  previous  Lemma,  consider 
the  function  v(x)  with  v(Q)  =  0  and  derivative  given  by 


v  (x) 
x v 


=  2A  x  -  2  $(s)ds 


;  0  <  x  <  1 


(6.  6) 


=  1  +  A  [e2(x'b)-l]  -  2  [Xe2(x's)<}>  (s)ds;  x  >  b 

b  l 


-v  (-x) 
x v  ' 


x  <  b . 


Proposition  6.2.  The  function  v(x)  defined  above  is  the 

(2) 

unique  (up  to  an  additive  constant)  solution  of  (6.2)  in  C  0R) , 
with  A  determined  along  with  the  constant  b  through  (6. 3) -(6.4). 
v(x)  is  also  the  smallest  nonnegative  function  satisfying 
equation  (6.2). 
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fb 

Proof.  From  (6.6),  v„(b  +  )  =  1  while  v  (b-)  =  2lb  -  2  |  4>(s)ds 

x  x  > 0 
by  (6.3).  Therefore  vx(x)  is  continuous  on  ]R.  On  the  other  han' 


Vxx(x)  = 


;o  <  x  < 


(6.7) 


=  2 


Xe2(x-b)  .<j>  (X)  -  2  fXe2(x's)0  (s)ds  ; 
_  }  b 


=  v  (-x) 
xx  ^  1 


X  >  b 

x  <  0 


is  clearly  continuous  on  IR.  From  (6.4)  and  the  fact  that  41  ( • ) 
is  strictly  increasing  on  IR+  one  gets:  vxx(x)  -  2(X-4>(b))  >  0, 
on  0  5  x  5  b,  as  well  as 


v 

XX 


(X) 


.00 

2  c'2(s'x)<t>  (s)ds  -  <f>  (x) 

J  Y 


>  0 ,  on  x  >  b . 


The  function  v(x)  is  even  and  strictly  convex,  therefore  minimal 
among  nonnegative  solutions  of  (6.2).  By  strict  convexity, 
0<vx(x)<l,on  0<x<b  and  v  (x)  >  1,  on  x  >  b.  It 
remains  to  verify  (6.2),  which  in  the  present  case  becomes 


(6.2)’ 


X  =  I  vxx(x)  +  ^ (x)  I  I X I  <  b 

=  j  vxx(x)  +  1  ‘  vx(x)  +  ^(x);  x  >  b 

=  \  vxx^x)  +  1  +  Vx)  *  *(x) ;  x  <  'b- 


(6.2)'  is  readily  verified,  by  substitution.  Uniqueness  of 
v^(x)  is  a  consequence  of  Lipschitz  continuity  of  the  function 
c(p)  defined  in  (6.1). 
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Proposition  6.3.  Suppose  that  A,v(x)  are  a  constant  a 

(2) 

CK  QR)  function,  respectively,  for  which  (6.2)  is  satisfied,  and 
such  that 

(i)  0  <  ^x(x)  <  1,  0  <  x  <  b 

(ii)  \(x)  =  1, 

(iii)  ^x(x)  >  1,  x  >  b, 

for  some  positive  constant  b. 

Then  the  function  v(x)  is  necessarily  strictly  convex, 
therefore  ^x(x)  is  strictly  increasing,  b  <  b  and 

(6.8)  A  >  A. 

Proof.  It  is  a  straightforward  exercise  to  verify  that  v^(x) 
will  necessarily  be  of  the  form  (6.6),  with  (X  , b )  replacing 
(A,b).  A  necessary  and  sufficient  condition  for  continuity  of 
v  (x)  is  then 

./ — j  ~~  fb  ' 

(6.3)  Ab  -  (Ks)ds  =  1/2, 

1  0 

while  (iii)  implies 

2 1  e*^x  ^^(y)dy  2  j  "  e  ^S<f>(b  +  s)ds 

(6.9)  A  >-£ - -  - - ,  all  x  >  b. 

e2(x-6)_1  1.e-2(x-b) 

A  necessary  and  sufficient  condition  for  (6.9)  is  (6.10)  below: 
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y*: 


(6.10) 


X  >  2  (  e  (b+s)ds . 
J0 


Indeed,  if  ^  <  2  f  e  ^s$(b+s)ds  holds,  then  (6.9)  is  eventually 

>  0 

false  as  x  -*  00 .  On  the  other  hand,  suppose  that  (6.10)  is  true; 
to  prove  (6.9)  it  suffices  to  prove 


(6.11) 

(1-e 

where  t  =  x 

-  b. 

-  2 1  • 


$(b  +  s)e  ^sds  >  [  <t>(b  +  s)e  ^sds,  all  t  >  0 
1  0 


e  2s[0(b  +  s)  -  4>(b  +  s-  t)]ds  >  0,  all  t  >  0, 


which  is  obviously  true  since  <P  ( •  )  is  strictly  increasing, 


Relations  (6.3),  (6.10)  arc  therefore  necessary  and  sufficient 
conditions  for  the  feasibility  of  (i)-(iii).  They  imply  that 
H(b)  <  0,  H(.)  being  the  function  introduced  in  (6.5).  But  H(.) 
is  strictly  increasing  so  b  <  b  and  therefore  X  >  X,  from  (6.3) 
and  (6.10).  .Strict  convexity  of  v(x)  is  proven  as  in 
Proposition  6.2,  Q.P.D. 


Once  the  solution  of  the  dynamic  programming  equation  (6.2) 
corresponding  to  the  smallest  possible  value  of  the  constant  X 
has  been  constructed,  we  proceed  to  prove  the  main  result  of  this 
section,  namely  the  optimality  in  the  class  &a  (Definition  5.2) 
of  the  law  u*(£)  =  a*(5  ),  with 


(.6. 12J 


a  (xj  -  -sgn  x,  |X|  >  d 

=  0,  I x |  <  b 


obtained  through  the  minimization 


(6.13)  a*(x)-v  (x)  +  | a  (x) |  =  min  [u-v  (x)  +  |u|]  =  c(v  (x)),  all  x  €  F 

iu|<l  x  x 


2c  ax 

-  m  ®  » 


Lemma  6 . 4 .  v(x)  =  0(e  )  ,  a  s  ]  x  |  -* 


Proof.  It  is  checked  that  for  all  x  large  enough 


.00 

vx(x)  =  l  -  X  +  2c2x  J  e'2y4>(y)dy  5  1  -  X  +  ji§ 


some  c  >  0.  The  result  follows  readily. 


Remark .  Dr.  Martin  Day  has  noted  that,  for  any  other  pair 
(A, 6)  as  in  Proposition  6.3,  the  functions  vx(x),v(x)  have  a 
growth  of  the  order  e2^,  as  |x|  ■+  <*>  .  I 


Theorem  6.5.  The  law  u*  £  'c*a ,  defined  through 


uj(0  »  a  (Ct),  all  5  €  C[0>T],  0  <  t  <  T, 

with  a*(0  as  in  (6.12),  is  optimal  in  ^a.  Furthermore 
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Proof .  Consider  any  law  u  £  &  and  the  Markov  process 
{ft,  ^,,^^,P^}  -solution  to  the  stochastic  differential  equation 
(4.4).  An  application  of  Ito's  rule  to  the  process  v(£“) ,  along 
with  (6.13)  and  equation  (6.2),  yields 


v(5“) 


v«  -  I  VO  *  us«>x‘0  *  *  |0V>s 

?  f'  I  VO  *  <*“*«»»  •  Js  *  {‘v5>s 

,tr  ~i  tt 

>  At  -  J  -KCg)  +  |us(CU)|  ds  +  j  Vx(C^)dws,  a.s.  (Pjj). 


Taking  expectations,  and  noting  that 


B«  [V^ds  <  Const. e2ct  (l  xl +tVJ  (V  '"^'ds  <  » , 

X  J  Q  x  s  -  X  J 


one  gets: 


E“v(^) 

(6.14)  •  x- 


_t_ .  nxi  + 1  Eu  |;[,(,U)  +  |Us(,u)l]ds >  A>  all 


From  (5.5)',  (5.7)  and  Lemma  (6.4)  one  gets 


oo 

im  E^v(^)  =  — 1—  v(y)dFu(y),  any  x  6  K. 

-►oo  X  Z  F u  (oo)  -00 


while  taking  (5.8)  into  account  and  letting  t  -*■  °°  in  (6.14) 


J(u)  >  X ,  any  u  e 


On  the  other  hand,  (6.14)  holds  as  an  equality  if  u  =  u  .  Therefore 


The 


The  last  two  relations  prove  optimality  of  u*  in  &  . 
u* 

density  of  5?  (•)  is  given  by 

P*(y)  =  (l  +  2b)  1 ,  |  y |  <  b 

(6.15) 

=  (l  +  2b)'1exp[-2(  jy| -b))  ,  |y|  >  b. 

7.  OPTIMALITY  OF  THE  LAW  u*  in 

In  this  section  the  performance  of  the  law  u*  of  Theorem  6.5 
is  compared  against  the  performance  of  any  admissible  nonanticipati 
control  law  u,  and  u*  is  proven  optimal  in  the  class  . 

The  method  consists  in  considering  the  finite-horizon 
optimization  problem:  minimize 

rT 

M>(0  +  lu  (S)  I  ]ds 
x  j  o  s  s 

subject  to  d£t  =  ut(C)dt  +  dw^,  Cg  =  x  and  u  €  .  Roughly 

speaking,  the  value  function 

fT 

V(x,t)  =  inf  E^  (♦(€_)  +  |  u  (4 ) | ]ds;  (x,x)  6  K  *  [0,T] 

u£ x  JT-t  s  * 

solves  the  Cauchy  problem 

(7.1)  VT  =  \  vxx  +  c(Vx)  +  -Hx);  (x,i)  €  JR  *  (0 ,T]  . 

(7.2)  V (x , 0 )  =  0  :  x  €  1R  , 

where  c(-)  is  the  function  defined  in  (6.1). 
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For  any  law  u  £  ,  Ito's  rule  gives 

rT 

Ex  J0[HCs)  +  K(C)l]ds  -  Vfx«T)> 

and  optimality  of  u  would  follow  if  it  were  proved  that: 

lim  =  X ,  all  x  £  IR. 

T+co  1 

In  the  remaining  of  this  section  we  justify  the  method  and  sub¬ 
stantiate  the  above  heuristics. 

Lemma  3.1:  A  priori  bounds  on  the  solution  of  the  Bellman 
equation  and  its  gradient.  Suppose  that  the  Cauchy  problem  (7.1), 

(7.2)  has  a  C2,1  solution  V(x,t)  on  1R  *  (0,T],  with  V(x,t) 
continuous  on  1  x  [0,T).  Then  the  following  inequalities  hold: 

(7.3)  V(x,t)  <  v(x)  +  At,  on  JR  *[0,T], 

(7.4)  |Vx(x,x)|  <  vx(|x|),  on  1R  *[0,T). 

I 

Proof.  It  is  immediately  verified  that  the  function 

M(x,t)  =  v  (x)  +  Xt  is  a  solution  in  1R  *  (0,Tj  of  the 

Cauchy  problem 

(7.5)  Mt  -  i  Mxx  +  c(Mx)  MW;  on  IR  x  (0,TJ 


(7.6) 


M(x,0) 


v(x)  ,  on  IR 
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>► 


and  that,  if 


i/  = 


is  the  parabolic  operator 

1  32  ,  3  3 

2  3x2  Vvx  3x  '  5T  ' 


then 


i/(M*N)  =  0,  in  ]R  x  ( 0  ,  T  ] 
M(x,0)  -  V(x,0)  =  v(x)  >  0,  on  JR. 


By  the  maximum  principle  (see  [3])  one  obtains  (7.3). 

Now  consider  a  sequence  (c  (p),  n  £  IN)  of  smooth  (piecewise 

2 

C  )  approximations  to  the  function  c(p),  with  cniP)  .  0  a.e.  on 
along  with  the  functions  V^n^(x,T),  M^n-*(x,T)  satisfying  (7.1), 
(7.2)  and  (7.5),  (7.6)  respectively,  with  c(-)  replaced  by  c  (•) 
Under  such  an  approximating  scheme,  V ^ (x , t ) ,  V^n^(x,t),  V^n^(x,t) 

X  XX 

converge  as  n  -*■  00  to  V(x,T),  V  (x,t),  V  (x,t)  respectively, 

X  XX 

uniformly  on  compact  (x,T)  sets.  Similarly  for  the  function 
M(x,T)  and  its  approximations. 

It  is  easily  checked  that  if  is  the  parabolic  operator 


(7.7)  if.  =  i 


is2  (  (V(n))  f  , 

1  3  .  c  (M^)  4-  +  -y  - jL.JLu(n) 

2  8?  n  x  3x  M(n).v(nP 

x  x 


■Vv 


XX 


3 

Tr 


then : 


y>y^-V(xnh  =  0,  on  IR+  x  (0 ,T] 


M^n)(x,0)  -  V^n)(x,0)  =  vx(x)  >  0,  on  1R+. 


It  can  be  shown  by  yet  another  application  of  the  maximum  principle 
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that 


Vxx)(x’T)  -  0 


on 


i„(Mx"))-i„(vxn)2  v(„) 


M 


(n) _v(n) 


xx 


1R  x  [Q,T].  Therefore,  the  potential  term 
is  nonpositive  on  1R  +  x  [0,T],  so  that  the 


the  strong  maximum  principle  is  applicable  (see  [3])  and  gives 
V^n)(x,t)  <  M^n)(x,x),  or  Vx(x,t)  <  Mx(x,t)  =  vx(x)  on  1R+  *  [0,T] 
in  the  limit  as  n  -*•  00 .  (7.4)  follows  since  V  (-,x)  is 
odd,  Q.E.D. 


Once  the  a  priori  bounds  (7.3),  (7.4)  have  been  established, 
one  can  apply  the  method  of  Theorem  VI  6.2  of  Fleming  and  Rishel 
[2]  to  prove  the  following  result: 

Proposition  7.2.  The  Cauchy  problem  (7.1)-(7.2)  has  a  unique 

2  1 

C  ’  solution  V(x,x)  on  1R  x  (0,T]  that  is  continuous  on 

1R  x  [0,T]  and  even  in  x. 

By  the  approximation  argument  used  in  the  proof  of  Lemma  7.1 
(or  directly;  see  [2],  F.xercise  VI. 9)  it  can  be  shown  that, 

Vxx(x’T)  -  0  in  m 

Consider  the  optimal  process  (n*)  for  the  finite  horizon 
problem,  defined  on  the  probability  space  (fi,  5*, P)  as  the  (strong) 
solution  of  the  stochastic  differential  equation 


(7. 

oo 

dn  t =  ^vxK'T-t))dt  +  dwt; 

0  <  t  <  x 

(7. 

.9) 

n0  =  x  >  0 

d 


2  3 


where  c(p)  =  - sgnp • 1 { |  | > 1 }  =  a*(p). 

Lemma  7.3.  For  any  x  >  0,  consider  the  stopping  time 

S  =  inf{t  <  t;  =  0} 

=  t,  if  >  0,  all  0  <  t  <  x. 

Then 

rs-  x 

(7.10)  Vfx.T)  =  E  <i>  (n  )dt. 

x  0  1 

Proof .  The  gradient  of  the  solution  to  the  Cauchy  problem 

2  1 

(7.1) -(7.2)  is  not  a  C  ’  function;  it  belongs,  however,  to  the 

2  1 

Sobolev  space  W  '  (D  *  [ 0 , T ] ) ,  for  any  p  >  1  and  any  bounded 

subset  D  clR,  and  satisfies  in  that  space  the  equation 

(VX)T  =  2  Cvx)xx  +  c(Vx)  (Vx)x  +  $(x)  on  1R  *  (0,T],  derived  from 

(7.1)  by  formal  differentiation.  For  functions  in  the  Sobolev 
space  a  generalized  Ito  formula  holds  (Zvonkin  [12],  Theorem  3) 
which,  applied  to  Vx(n^,x-t)  on  [0,S]  along  with  (7.8^  and  the 
fact  that  V  (n^,T-S)  =  0,  a.s.,  yields  (7.10),  Q.E.D. 

X  b 

Consider  now  the  "optimal  process  (£t)  for  the  stationary 
control  problem": 

t7-11)  d^t  =  ^(vxUj))dt  +  dwt,  t  >  0 


2] 
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defined  on  the  same  probability  space  and  with  the  same 

initial  condition  as  for  (7.8),  (7.9). 

Lemma  7.4.  |5*|  <  |  (  <  |x  +  w  |,  0  <  t  <  x;  a.s.(P). 

Proof.  An  easy  consequence  of  the  comparison  theorem  of  Ikeda 
and  Watanabe  [6]  and  (7.4)  of  Lemma  7.1. 

From  (7.10)  notice  that,  for  any  x  >  0,  V  (*,x)  increases 
to  infinity  as  x  -*•  « ,  since  <}>(•)  does.  Therefore,  for  any 
x  +  0 , 

(7.13)  s(x)  *  max{x  >  0;  Vx(x,x)  =  1} 

is  well-defined  and  finite. 

Lemma  7.5.  s(x)  is  left  continuous  and  decreasing  on  Bt+ . 

Proof.  It  can  be  checked  that  for  the  approximating  functions 
introduced  in  the  proof  of  Lemma  7.1:  “*1  (V^^ )  =  0  in  |R  *  ( 0 ,  T  ]  , 

being  the  operator  defined  in  (7.7),  and  vfj^(x,0)  =  <j>  (x)  >  0, 
on  B1  .  By  a  maximum  principle  argument:  ^xxJ*-x,T^  -  ®  on 
1R+  x  [0,T],  and  therefore  ^X^X»T2^  -  ^x^x,Xl^’  ®  -  T1  <  x2’  x  -  ® 
in  the  limit  as  n  -»  This  proves  the  monotonicity  of  s(’).  Lef 
continuity  is  an  easy  consequence  of  definition  (7.13)  and  monotonic 

Lemma  7.6.  lim  \\  (x,x)  =  v„(x),  uniformly  on  compact  x-sets. 

X-*.oo  *  x 


Proof. 


Notice  that 


<VVxV  7  <YVx>x*  *  ;<V<YVx  *  -  ‘<Y> 


;  7  <YVx>xx  *  c<V<YVx' 


on  1R  *  (0,T],  by  convexity  of  V,  monotonicity  of  c  and  (7.4). 

a  £  * 

An  application  of  the  generalized  Ito  formula  to  v  (£  )  -  V  (S  ,  x - 1 

X  L  Xu 

gives : 

0  5  V*)  -  Vxtx'T)  ?  E  YV  ■  lRlr)\l(‘)dP’ 

where : 


R  =  inf { t  <  x;  S  =  0} 


=  x,  if  S  >  0,  all  0  <  t  <  x 


We  note  that:  E  V*  +  \|S*|)  - ** 


x  t» 


Vx+<5Uyl  )P*Cy)Jy  <  °°  as  long  as 


0  <  <5  <  —  -  1,  by  virtue  of  (5.5)  and  (6.15).  So 

l  +  6i*| 

sup  Ev  (IS  I)  <  » ,  which  implies  uniform  integrability  (and  hence 
x>0  x 

also  absolute  continuity  with  respect  to  measure  P)  of  the  family 
of  random  variables  (  vx (  K  *  I  ) )  x>0’  t^ie  ot^er  hand, 


P(R  =  x)  <  p(x  +  w  >  0,  all  0  <  t  <  x)  =  24>(xx'1/2)  -  1  - 


as  x  -*■  »,  uniformly  on  compact  x-sets;  see  Gihman  and  Skorohod 
[4;  §1].  The  result  follows  by  uniform  absolute  continuity. 
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Corollary .  s(t)  1  b,  as  t  +  ® , 


Proposition  7.7.  lim  — * 'x » ^  =  A,  any  x  €  1R. 

T-k» 

V  (  Y  I  ^ 

Proof.  That  lim  sup  — — * — -  <  A,  uniformly  on  compact  x-sets,  is 


T+oo 


consequence  of  (7.3).  To  prove  the  opposite  inequality  note  that, 
by  virtue  of  Lemma  7.4, 


V(x,t)  =  E 


>  E 


1 4> (n  )  +  1  ]dt 

o  {|  nt  |  >S  (T-t)  > 


+  1  *  ]dt 

0  (  Ktl>s(x- t)} 


and  there  fore,  for  any  x  G  1R, 


V.UrU  >  i  f: 

T  T 


mst)  +  1  * 

0  1  {|C*hb> 


(7.14) 


h.xl*"-'”  '  Ft,x<b>  -  {Pt,Xf-s(T-t^ 


-  ,;t,x(-^]ldt 


where 


Ft  x (y )  =  Pttt  <  y  K0  -  x)  - F  (y)  = 

’  t+oa 


p*(z)dz, 


F:*(-)  is  the  ergodic  probability  distribution  function  correspondin 

.  * 
to  the  optimal  law  u  in  &a . 


Now 
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(7.15) 


im  y  [  [  pt  (s(T-t))  -  P  (b) 
+oo  J  n  *■  >*  t  »x 


lim 

T+» 


-  {  F*  (s  (T-t) )  -  F*(b))]dt  =  0. 


Indeed,  the  integrand  in  (7.15)  is  dominated  by 

2  sup  |  F ..  (y)  -  F*(y)|,  which  tends  to  zero  as  t  -*■  00 ,  because 

y€3R  z  ’ x 

*  c  * 

F  is  (absolutely)  continuous  and  F  — y  F  (sec  [1],  p.  25 

L  ,  X 

Ex.  8.1.13).  Hence 


lim  y 

T-w° 


X 


(F 

0 


t,x 


(5(T't)) 


F  (b))dt  =  lim  ~ 
x-*® 


fT  * 

F  (s(x-t))  - 

0 


F*(b))dt  =  0 


*  * 

since  lim  F  (s(t))  =  F  (b) ,  by  the  Corollary  to  Lemma  7.6.  By 

the  same  token,  the  entire  second  term  on  the  right  hand  side  of 
(7.14)  converges  to  zero  as  t  ■>  « ,  while  the  first  term  converges 
to  A.  Therefore,  for  any  x  E  IR : 


lim  inf  ^  -X^-T  >  A,  Q.E.D. 

T  ->oo  ^ 


We  are  now  in  a  position  to  prove  the  main  result  of  this 
section . 


Theorem  7.8.  The  law  u  of  Theorem  6.5  is  optimal  in  the 
class  %  of  admissible  nonanticipative  controls,  i.e.  for  any 
u  €  %  ,  x  E  JR: 

(7.16)  J(x,u)  =  lim  inf  ±  E^  [  [<t*(0  +  |u.(C)]]dt  >  X  =  J(u*). 

T-wo  1  x  Jo  z  Z 
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Proof.  Take  any  law  u  €  %  along  with  the  Girsanov  solution 
process  (5^)  satisfying  =  ut(5u)dt  +  dwt ,  =  x  on 

(SI,  as  in  Section  2,  and  apply  Ito's  rule  to  the  process 

V(£^,T-t),  V (x , t )  being  the  function  of  Proposition  7.2: 


V (x,T)  =  V (£g>T)  -  V(^,0)  =  -jo[ut(5U)Vx(^,T-t) 


+  iVxx(HJ‘t)  *  Vr(S'T-t)]dt  -  Vx(^,T-t)dwt. 


Because  c(p)  =  min  (up+|uj),  we  get 

I  u|  <1 

T  X 

(7.17)  V  (x,T)  <  |o[<f(^)  +  |ut(5U)|]dt  -  |oVx(C^T-t)dwt  a.s.  (P“) , 

any  x  €  JR,  T  >  0. 


The  expectation  of  the  stochastic  integral  on  the  right  hand  side  of  (7.17) 
is  zero,  because 


EU  f  V2£“  T-t)dt  <  Eu  f  v2(^)dt  <  const 
xj0  x*t’  '  x|0x4J  - 


.  e2»(|x|.T)  f^K1^  < 
J0  l 


by  virtue  of  (7.4),  and  it  follows  from  (7.17)  by  taking  expectations  that 


T 

5  T  Ex  |  [♦(«?)  +  lut^U)  l]dt;  any  x  €  1R,  T  >  0. 


(7.16)  is  obtained  by  a  passage  to  the  limit  as  T 
into  account  the  assertion  of  Proposition  7.7. 


00 


and  taking 
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